20 Apr 2023

What is an Analytical Pipeline?

What are the issues?


  • Lots of manual steps
  • Hard to reproduce
  • Mistakes are easily made and hard to track
  • The steps aren’t recorded
  • Using multiple independent tools
  • How do we keep track of which file versions people have?

What is a Reproducible Analytical Pipeline (RAP)?

Source: The Turing Way



  • It is easily repeatable
  • It is easily extendable
  • It is automated
  • It minimises mistakes
  • It is fast
  • It builds trust

Image taken from: The Turing Way book

What does a RAP look like?

What are the benefits?

Source: The Turing Way


  • Easy for others to use
  • Others can change and adapt
  • All steps are recorded
    • Including whilst it is built
  • Automated and fast
  • Open and promotes trust

Image taken from: The Turing Way book

What do we mean by reproducible?

Source: The Turing Way


We want to look back and be able to repeat our work easily and quickly.

What are the benefits?

  • Helps build trust
  • Not reliant on single individual
  • Can be adapted and re-used

Image taken from: The Turing Way book

We don’t have to do it all at once



The building blocks of a RAP:

  • Version control
  • Using open-source tools
  • Create reproducible code


… are useful in their own right, each will improve the auditability, speed and quality of your work.

Sheffield City Council RAP

The Council’s BI Team have used some of the RAP building blocks. PAS and the BI Team are now further investigating the potential of RAP:

  • The Government’s Analysis Function and the NHS are fully committed to RAP, but how applicable is it to Local Government?
  • Evidence of RAP benefits and costs?
  • How could RAP evolve alongside current processes and tools?
  • Planning the delivery of a series of short internal webinars on scripting, version control, and RAP.
  • Considering re-engineering a current pipeline as a RAP pilot.
  • Discussing a version control platform with IT.

Further RAP resources

SCC RAP investigation Kanban

Citing The Turing Way

Many of the images used in this presentation were taken from The Turing Way book.

Full citation:

The Turing Way Community, Becky Arnold, Louise Bowler, Sarah Gibson, Patricia Herterich, Rosie Higman, … Kirstie Whitaker. (2019, March 25). The Turing Way: A Handbook for Reproducible Data Science (Version v0.0.4). Zenodo. http://doi.org/10.5281/zenodo.3233986